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ABSTRACT 

Two studies are reviewed to determine their relevance 
to proposed legislation (H.R. 2620) concerning athletic scholarships. 
The review was conducted at the request of U.S. Representative James 
J. Howard who proposed the bill. H.R. 2620 would require colleges to 
graduate at least 75% of their scholarship athletes within 5 years in 
order for contributions to their athletic departments to qualify as 
tax deductible. The two studies are: (i) "National Collegiate 
Athletic Association Survey of Graduation Rates After Five Years for 
Males First Entering College in Fall 1975," prepared by the American 
College Testing Program; and (2) "Study of Freshman Eligibility 
Standards Technical Report," prepared by Advanced Technology, Inc. 
For each study, attention is directed to the characteristics of the 
sample, the quality of the data, and data analysis. It is concluded 
that both studies have limited relevance for the legislation. The 
bill is directed at athletes who remain at the same college and 
receive athletic scholarships for 3 years. Both studies reported 
graduation rates for athletes and nonathletes only for freshman 
college entrants rather than for those who had been enrolled for 3 
years. Serious methodological problems were also found in each study. 
(SM) ' 
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The Honorable James J„ Howard 
House of Representatives 

Dear Mr. Howard: 

Subject: Review of Two Studies on College Athlete 
Graduation Rates (GAO/PEMD-85-9 ) 

This report responds to your June 26, 1985, request for a 
GAO review of two studies on college athlete graduation rates. 
You asked us to "determine whether or not the methodology used 
was accurate and whether or not these studies are relevant" in 
discussing your proposed legislation H.R. 2620. Your bill would 
require colleges to graduate at least 75 percent of their 
scholarship athletes (defined as students who receive athletic 
scholarships for at least 3 academic years) within 5 years in 
order for contributions to their athletic departments to qualify 
as tax deductible. 

The two studies we reviewed are (1) "National Collegiate 
Athletic Association Survey of Graduation Rates After Five Years 
for Males First Entering College in Fall 1975," prepared by the 
American College Testing Program in April 1981 (we refer to this 
study as "ACT"), and (2) "Study of Freshman Eligibility Standards 
Technical Report," prepared by Advanced Technology, Inc., in 
August 1984 (we refer to this study as "AD TECH"). ACT concluded 
that male athletes graduated at a rate equal to or higher than 
that of male nonathletes. AD TECH concluded that student athletes 
graduated at about the same rate as students in general. Our 
review of these two studies was based on professionally accepted 
methodological standards for behavioral research and, in the 
interest of timeliness, limited to information provided in the 
two written reports. 

We found that both studies have limited relevance for your 
legislation. H.R. 2620 is directed at athletes who remain ac the 
same institution and receive athletic scholarships for 3 years. 
Both studies reported graduation rates for athletes and 
nonathletes only for freshman college entrants rather than for 
those who had been enrolled for 3 years. In addition, ACT die* not 
report analyses on athletes by scholarship status. The AD TSCH 
researchers conducted analyses by grant-in-aid (scholarship) 
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status although they included some athletes at institutions that 
do not offer athletic scholarships. 

We found serious methodological problems in each study. 
First, genaralizaoility : the characteristics of the samples 
signify that generalizations can be drawn from each study only to 
the schools that participated in that study. Second, data 
quality: the poor quality of the data in ACT limits the 
confidence that can be placed in the study's results and the 
precision with which the results can be interpreted. Third, data 
analysis: inappropriate data analyses in both studies mean that 
the reported findings cannot be used to compare the graduation 
rates of athletes and nonathletes. More appropriate and 
meaningful analyses could be performed on the data collected for 
both ACT and AD TECH, provided that these data have been 
retained. We believe the AD TECH data are of sufficiently high 
quality to allow confidence in the results of such analyses. 

More specific details are provided in the enclosures to this 
report. Our review of ACT is in enclosure I, of AD TECH in 
enclosure II. Enclosure III contains a hypothetical illustration 
of what we mean by "appropriate" and "inappropriate" data 
analyses . 

The prrblems we found are such that the two studies are 
inconclusive for the purpose of answering your questions. AD 
TECH was designed primarily to assess the effects of instituting 
eligibility standards for participation in athletics in the 
freshman year, and we did not review its relevance or quality for 
this purpose. As the two studies stand, however, they do not 
provide a basis for concluding that the graduation rates of 
college athletes are higher or lower than, or the same as, those 
of all other college students. 

As requested by your staff, we did not send a draft of th"» s 
report for advance review to the American College Testing 
Program, Advanced Technology, Inc., and the National Collegiate 
Athletic Association, in the interesL of timeliness. No further 
distribution of this re.>rt will be made for 2 days from the date 
of this letter. At that time, we will send copies to these three 
organizations and will make additional copies available to others 
who are interested. 



Sincerely yours, 




Eleanor Chelimsky 
Director 
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REVIEW OF ACT 



1. The characteristics of the sample limit the conclusions that 
can be drawn from this study to men at 46 member institutions 
of the National Collegiate Athletic Association (NCAA). 

* ACT appropriately chose to take a random sample of athletes 
and nonathletes to allow generalization of the results to 
all NCAA institutions. The researchers sent questionnaires 
on student graduation rates to a random sample of 200 mem- 
ber institutions of NCAA. However, only 46 institutions 
responded. This response rate, 23 percent, is low by pro- 
fessional standards. 

* ACT asked responding Institutions to report graduation 
rates for men only. 

* ACT did not establish that the 46 responding institutions 
were typical of NCAA member institutions in general. The 
low response rate upset the initial design, which called 
for a random sample for drawing general conclusions about 
NCAA member institutions. ACT did not report standard 
analyses to demonstrate that participation in the study was 
unbiased. Thus, the institutions that participated may 
differ from other institutions on factors such as institu- 
tional size, prestige, or NCAA division, which could affect 
reported outcomes, or they may differ from nonparticipants 
directly in graduating a higher or lower percentage of 
athletes. 

2. The quality of the data in this study is low, by professional 
standards, for placing confidence in comparisons of athletes 
and nonathletes. 

* The data may contain reporting bias. The respondents for 
each institution surveyed were largely directors of ath- 
letic departments who, it could be argued, may have been 
predisposed to report more favorable information for 
athletes than nonathletes. Although there is no proof of 
bias, ACT did not report any efforts to minimize its 
likelihood, such as independent verification of reported 
data. Thus, differences between athlete and nonathlete 
graduation rates could in fact be less favorable to 
athletes than those reported. 

* Some of the students classified as nonathletes may partici- 
pate in sports as frequently as some of the students 
classified as athletes. The survey instructions did not 
clarify how status as an athlete was to be defined. The 
definition was left to the discretion of the respondents, 
who may have used different criteria for classifying 
athletes and nonathletes. In addition, classification was 
based on status upon college entrance. 
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Some students may have changed status during the first year 
or may have been misclassif ied at the outset of the study. 
ACT did not report information on the reliability or sta- 
bility of classifications. Therefore, confidence in the 
reported contrasts between athletes and nonathletes is less 
than it might have been. 

3. Inappropriate data analyses in this study limit conclusions. 

* The overall rates of graduation of athletes and nonathletes 
reported in ACT do not reflect the typical pattern in the 
46 institutions surveyed. The overall rates reported in 
this study are biased toward the rates of graduation at the 
largest institutions in the sample. ACT did not provide 
information on the range or distribution of rates across 
institutions and did not report analyses to explain the 
large differences between o^eiall and median rates. 

The data suggest that a few of the larger institutions in 
the survey had relatively high rates of graduation for all 
students (athletes and nonathletes) compared to the other 
institutions surveyed and that the rates for athletes were 
exceptionally high at a few institutions. Therefore, the 
overall rates nay seem higher than actual rates in most 
schools for both athletes rind nonathletes. 

* The median rates of graduation for athletes and nonathletes 
reported in ACT do not reflect the typical pattern of dif- 
ferences in the 46 institutions ^^rveyed. (The median is 
the point at which half the schools graduated students at 
higher rates and half at lower rates.) 

There is no problem with using median rates to describe the 
central tendency of a single group or to compare rates in 
two groups when the groups are not matched on a relevant 
characteristic. There jls a problem witn using two separate 
group medians to compare athletes and nonathletes matched 
by institution of attendance. In ACT, the median rates 
indicate only that 23 of the institutions graduated non- 
athletes at rates lower than 33.8 percent and 23 graduated 
athletes a*- rates lower than 36.9 percent. These are not 
necessarily the same 23 institutions. An appropriate pro- 
cedure is first to calculate the difference in the rates of 
graduation at each institution and then to find the median 
of this difference. 

* ACT did not report appropriate analyses for establishing 
the typical pattern of difference between athlete and non- 
athlete graduation rates within the 46 institutions 
surveyed. The average difference in graduation rates for 
athletes and nonathletes in these 46 institutions may be 
zero or it may favor athletes or nonathletes* There is no 
way to determine this from the analyses that were reported. 



ENCLOSURE II 



ENCLOSURE II 



REVIEW OF AD TECH 



1. The characteristics of the sample limit the conclusions that 
can be drawn from this study to black and white athletes at 
206 Division I member institutions of: NCAA. 

* AD TECH decided to survey all 276 NCAA Division I member 
institutions, which is one appropriate way to draw adequate 
conclusions about differences in graduation rates for this 
type of institution. The response rate, 206 of 276, or 

75 percent, is reasonable by professional standards. 

* However , the participating institutions are not fu lly 
representative of NCAA Division I member institutions. 

The participating institutions had larger fall 1982 enroll- 
ments, more were public than private, and more offered a 
doctorate or first professional degree than Division I 
institutions as a whole. Rates of participation by NCAA 
subdivision and NCAA district varied. Differences between 
participating and nonparticipating institutions on any of 
these factors could imply differences in graduation rates 
as veil. 

* AD TECH reported results only for black and white athletes. 

* The intended sample ot grant-in-aid athletes included an 
unknown number of athletes who did not receive grants-in- 
aid. An unreported number of participating institutions 
did not provide athletic grants-in-aid. These institutions 
provided information on a sample of their athletes. AD 
TECH included these data in the overall analyses of the 
performance of grant-in-aid athletes. Therefore, confi- 
dence in drawing conclusions from the study about grant- 
in-aid athletes is less than it might have been. 

2. The quality of the reported data in this study appears to be 
high . 

* Data collection was coordinated by institutional liaisons 
designated by each institution ' s chief executive officer, 
and in some institutions as many as five departments parti- 
cipated in data-reporting. Answering the survey required 
access to formal student records for information on 
athletes . 

* The definition of "athlete" for most institutions was a 
student receiving an athletic grant-in-aid. For military 
academies and other institutions that did not award 
athletic grants-in-aid, "athlete" was defined as a student 
participating in NCAA-recognized sports. 

* Quality-control procedures were used to minimize errors in 
preparing the data for analysis. 
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3. No appropriate analysis was reported to support a contrast 

between athletes and students in general on graduation rates. 

* How the graduation rates of athletes compared to those of 
nonathletes in the 206 participating institutions is 
unknown, AD TECH reported overall graduation rates for 
athletes and median institutional graduation rates for 
students in general. These two statistics cannot be 
meaningfully compared. 
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HOW STATISTICS CAN BE MISLEADING 



We constructed hypothetical data to illustrate how the use of 
different summary statistics can lead to different conclusions. 
In this example, four hypothetical schools A-D are equal in size 
and School E has a much larger student body and a larger number of 
athletes. School E is the only school that graduates athletes at 
rates greater than it graduates nonathletes. Tables 1 and 2 show 
these hypothetical data. Table 3 shows appropriate and inappro- 
priate analyses and explains why the ACT and AD TECH analyses were 
inappropriate . 



Table 1 

Data on Graduation Rates of Individual Students 
at Five Hypothetical School^ 



No. of nonathletes No. of athletes 



School 


Students 


Graduates 


Students 


Graduates 


A 


500 


400 


50 


20 


B 


500 


400 


50 


20 


C 


500 


150 


50 


0 


D 


500 


150 


50 


10 


E 


2,000 


500 


200 


150 


Total 


4, 000 


1,600 


400 


200 


% graduating 
(overall 


1,600/4,000 = 40% 


200/400 


= 50% 



graduation 
rate) 

Group with higher rate: Athletes 
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Table 


2 




Data 


on Graduation Rates of 


Nonathletes and 


Athletes 




in 


Five Hypothetical Schools 






Graduation rate 


Difference 


Group with 


School Nona th let es 


Athletes 


in rate 


nigncr rate 


A 


80% 


40% 


40 


lNcvnciun ictco 


n 
]_> 


ftn 

o w 


40 




Nonathletes 


C 


30 


0 


30 


Nonathletes 


D 


30 


20 


10 


lNVJil dun IcLco 


E 


25 


75 


-50 


Athletes 


Number of 
schools 


5 


5 


5 




Graduation 
rate 
Mean 
Median 


49% 
30 


35% 
40 


14% 
30 a 





a The mediae difference is computed vertically, not horizontally. 



Table 3 



Possible Comparisons from the Hypothetical Data 



Statistics for 2 groups 
from the same 5 schools 

Graduation rate 
Median 
Overall 
Mean 



Athletes Nonathletes 



40% 

50 

35 



30% 

40 

49 



Group with 
higher rate 



Athletes 
Athletes 
Nonathletes 



Difference in graduation 
rate 
Mean 
Median 



14% 

30 



Nonathletes 
Nonathletes 



Percentage of schools 
graduating students 
at higher rates 



20% 



80% 



Nonathletes 
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Median rates are appropriate summary statistics for one group 
or for two independent groups. In this example, the two groups are 
not independent because the samples of athletes and nonathletes 
were drawn from the same schools. In this kind of situation, it is 
inappropriate to compare two medians. Such a comparison provides 
misleading information about the typical case. 

Overall rates are appropriate for answering questions about 
aggregate resu Its (all athletes and nonathletes ) . They are inap- 
propriate for answering questions about the typical case, because 
they ignore the information that students come from different 
schools. Thus, overall rates are biased toward the rates at insti- 
tutions with the largest number of students. In this example, the 
difference between overall rates is largely a consequence of the 
unique pattern at School E. 

Mean rates are appropriate for answering questions about the 
typical case. They can be misleading when interpreted alone, 
because they are sensitive to extreme values. Thus, it is inappro- 
priate to draw inferences from means in the absence of other infor- 
mation. 

The mean difference in graduation rate is algebraically equiva- 
lent to the difference in mean rates and is therefore interpreted 
in the manner described above for mean rates. 

The median difference in rates is always appropriate for 
answering questions about the typical case. It is not sensitive to 
extreme values or institutional size. The median difference is not 
equivalent to the difference in medians. 

The percentage of schools showing a particular outcome is 
also an appropriate statistic. It provides information not on the 
size of the typical difference between two groups but only on the 
pattern of difference across schools. 

Finally, it is always inappropriate to compare two different 
statistics to draw inferences about group differences. Each sta- 
tistic is calculated in a different way and each has a different 
meaning . 

AD TECH compared two different statistics — median institu- 
tional graduation rates of students in general and overall gradua- 
tion rates of athletes. 

ACT compared median institutional rates for athletes and non- 
athletes as well as overall rates for both groups . 



ERLC 



